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ABSTRACT 



Since 1991-92, student writing portfolios have been part of 
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continuously improving or continuously declining writing portfolio scores led 
to identification of 36 indicators of successful writing programs. These 
indicators were used to develop a comprehensive needs assessment instrument, 
to be used in a school self-evaluation involving all stakeholders. This paper 
describes the pilot test and field test of the instrument, the School Study 
of Writing Instruction. The pilot test was carried out in a small K-6 school 
in western Kentucky and was deemed successful due to faculty participation 
and sense of ownership and the steering committee's customizing of the 
instrument. The results generated 18 recommendations and 22 proposed 
revisions. The field test was conducted in 11 primarily rural and small-town 
schools. The revised instrument involved a process in four steps: 
interviewing administrators, students, and teachers using questions related 
to the 36 quality indicators; writing a report profiling the school; rating 
the school on the indicators; and setting priorities for improvement. Five 
schools were assigned an external facilitator. Overall, conclusions about the 
self-study process were positive. Most schools completed the study to a 
satisfactory point and believed the effort to be worthwhile. The time 
required v/as a concern, but facilitator assistance was highly beneficial. The 
instrument was generally valid and reliable, but may be improved with various 
adjustments. Appendices contain statistical data tables and a time and task 
summary. (Contains 30 references.) (SV) 
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INTRODUCTION 



Background on the Kentucky Education Reform Act 



Writing portfolios are a critical component of writing instruction in Kentucky, where 
education reforrii is driven by a high stakes performance-baSed'assessment. The architects of the 
Kentucky Education Reform Act (KERA), perhaps the most extensive state-level education reform 
ever initiated, mandated this new assessment format in order to encourage changes in teachers’ 
instructional practices as one means of improving education in the state. Although some educators 
would disagree with the premise that assessment drives instruction, KERA is very much based on 
that premise, as demonstrated both by the new assessment design and by the school-level rewards 
and sanctions that are largely determined by student performance on the new tests (Kifer, 1994). 
Evidence from early studies involving several rural Kentucky districts indicates that the reform is 
indeed impacting instruction and the impact is largely due to the design and accountability weight 
of the assessments (AEL, 1994). 

Another part of the reform that also aims to impact instructional practices is the increased 
funding of professional development for teachers. The reform designers recognized that changing 
instruction would necessarily go hand-in-hand with intense professional development efforts. In 
order to support this substantial need, legislators increased professional development funds from the 
level of $1 per student at KERA’s inception to the 1999 level of $23 per student. Professional 
development options have not only increased significantly, but many focus directly on helping 
teachers attend to the demands of the new tests (AEL, 1996). Professional development in writing 
is provided largely through the state writing program. The Kentucky Writing Program is strongly 
influenced by the National Writing Project, which began in northern California in the mid-1970s. 
Eight imiversity -based summer institutes using the National Writing Project model have been 
available in Kentucky since before KERA, and other state professional development in writing is 
infused with understandings about the process of writing from that model. 



For those imfamiliar with Kentucky’s accountability system, a brief description follows. A 
school’s accountability index is largely determined by student scores on the open-response sections 
of the test and the writing portfolio. Student scores on science, reading, mathematics, and social 
studies, and scores from the writing portfolio section of the test comprise the great majority of the 
accoimtability index. In 1991-92, the first year of the assessment, the writing portfolio formed the 
entire writing index. Starting in 1996 and continuing, scores from the on-demand writing section 
of the test were included in the writing score, but the on-demand section still determines only a small 
portion of a school’s writing index. Therefore, from the beginning of the assessment, the writing 
portfolio scores have com.prised a significant portion of the accountability index. 

For this discussion, it is also important to note that rewards and sanctions are determined by 
comparing an individual school’s performance to its own past performance. In order to receive 
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rewards, a school must demonstrate sufficient improvement over the past year’s performance. 
Sanctions in the form of additional technical assistance and dollars occur when schools fail to 
improve sufficiently or demonstrate lowered student performance. 

The writing portfolio assessment has been in place for eight years. Over those years, 
personnel from the Kentucky Department of Education (KDE), especially those involved in the 
Kentucky Writing Program, have noted some obvious patterns in individual school scores. Some 
schools have demonstrated continual improvement in their writing scores over the years, while other 
schools’ writing scores have fluctuated. This study was initiated in order to identify indicators of 
successful schools’ writing programs and to use that information in the development of professional 
resources. 



Research on Writing Program Indicators 

The research literature offers little guidance in the search for school-level writing program 
indicators. Lipson and Mosenthal, in a paper presented at the 1997 AERA conference, looked at 
indicators by the instructional style and pedagogical stance of individual teachers in relation to 
Vermont’s writing portfolio assessment. They reported that the writing portfolio assessment 
changed how the teachers in their study taught writing, but that the amount and type of change 
depended on the perspective of the teacher. A conclusion of the study, which included surveys, 
observations, and interviews, was that “the diversity of teachers and the variety of their teaching 
contexts must be taken into account” by policymakers attempting to influence practice (Lipson and 
Mosenhal, 1997, p.l4). 

While the Lipson and Mosenthal study attempted to determine indicators that influence 
individual teacher practice, Rand studies in both Vermont and Kentucky focused on implementation 
at the state level (Koretz, et ah, 1992, 1993, and 1996). Teacher surveys revealed that even though 
a significant portion of teachers reported that the assessment was overly burdensome, changes in 
instruction varied widely among teachers. However, none of these studies examined school writing 
programs or attempted to determine indicators that differentiate more successful from less successful 
schoolwide programs. Other researchers have identified state-level indicators of enacted curriculum, 
and of mathematics and science instruction in particular. They offer guidelines for determining and 
using such indicators (Blank, 1993, 1997; Porter, 1991; Smithson, 1994). The literature is silent, 
however, on the subject of writing program indicators. An ERIC search produced no documents that 
addressed the topic. 




Kentucky Writing Program Research Project 
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One of the mandates of KERA of 1990 is that every student develop a writing portfolio for 
which he/she is accountable at the 4“’, 7"’, and 12'*' grades (Kentucky Department of Education, 
1994b, 1996, 1999). There are four benchmarks of achievement by which writing portfolios are 
judged: novice, apprentice, proficient, and distinguished. Schools are required by KERA to achieve 
an average of proficient on students’ writing portfolios by 2010. However, since the inception of 
KERA, teachers have found it possible to move students from novice to apprentice, but more 
formidable to maneuver them to the next benchmark of proficient. Presently, many students at 
varying degrees of mastery have been appraised at the apprentice level. Some teachers have voiced 
concern that the range of mastery within the apprentice benchmark is too wide. They feel that many 
students deserve to be evaluated at a higher level, and to adequately accommodate their range, 
intermediate benchmarks between apprentice and proficient should be created. 

In 1996, the KDE and Appalachia Educational Laboratory (AEL), Inc. began a collaborative 
project to assist teachers in writing instruction and ultimately, to improve their students’ writing 
portfolio scores (AEL, 1995, 1997a). The project, titled the Kentucky State Project: Designing 
Professional Development for Portfolio Improvement, shall be hereafter referred to as the Kentucky 
writing project or project. 

Prior to the inception of the project, approximately 1 00 Kentucky schools were identified as 
having shown consistent gains in writing portfolio scores between 1991-1994 (AEL, 1995). 
Furthermore, although some have criticized the writing portfolio scoring process as being subjective, 
scoring audits showed that scores in these schools were more reliable than scores in less successful 
schools. To help less successful schools achieve these same gains, AEL and KDE were interested 
in a preliminary study to identify correlates of score gains in successful schools, possibly followed 
by the creation of new professional development materials and strategies (AEL, 1995). As stated 
in the project proposal, the original objectives of the research and development effort were (1) to 
establish correlates to gains in writing scores; (2) based on these correlates, to develop and field test 
new professional development materials to improve both teacher writing instruction and accuracy 
in scoring student portfolios, implement the new materials in a training program, and disseminate 
the materials statewide and nationally; and (3) to determine the applicability of the new professional 
development materials to content areas other than writing instruction. 

The Kentucky writing project was conceived as an applied research and development effort. 
In the first meeting between KDE and AEL, the original objectives and activities of the project 
changed substantially (AEL, 1998a). Focus converged on the first objective, that of identifying 
correlates of writing score gains, later termed quality indicators or indicators. The second and third 
objectives of designing new professional development were put on hold, because key decision 
makers at KDE agreed that preliminary research into correlates of gains required in-depth attention. 
First, KDE staff proposed that advancing students to the proficient level requires a completely 
different set of teaching strategies than those that had been sufficient for getting students to the 
apprentice level (AEL, 1998a). Second, KDE staff hypothesized that perhaps other variables in 
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addition to professional development and teaching strategies were decisive factors in successful 
schools’ continual improvement. Third, it was felt that the professional development already being 
provided by KDE was of good caliber: it was schools’ level of participation in, not the content of, 
these programs that differentiated schools’ writing programs. For these reasons, KDE decision 
makers were wary of prematurely designing new professional development materials as a quick fix 
for declining schools. 

The initial proposal stipulated that interviews with various school stakeholders would be 
performed in a few schools to develop appropriate interview protocols and to begin the process of 
identifying correlates of gains in writing scores (AEL, 1995). Once the interview protocols were 
refined, a broad telephone survey of successful and unsuccessful schools would be conducted in 
order to establish these correlates of score gains (and conversely, the correlates of low or declining 
scores). However, the idea of a telephone survey had been dismissed early, also at the first KDE - 
AEL meeting, in favor of a more penetrating scrutiny. Key KDE decision makers felt that the 
examination needed to be more in-depth and continue to include the insights of the various 
stakeholders. AEL staff agreed with these recommendations and encouraged the inclusion of 
students as one of the stakeholder groups. A pervasive, in-depth examination could not be 
effectively accomplished by means of a telephone survey. Therefore, a series of site visits was 
plarmed instead for the purpose of identifying the correlates, or quality indicators. 

At the second meeting, the KDE Writing Program staff, the KDE Regional Writing 
Consultants (then supervised by KDE, now by regional service centers), and various staff of AEL 
formed a collaborative research team, also referred to as the research team or team. The 
collaborative research team wrote interview protocols in preparation for site visits, and then refined 
them based on their use in exploratory visits to seven schools. 

Based on a series of site visits to 22 of 43 continuously improving schools, as well as the 
initial exploratory visits, data from KDE, and a review of relevant literature, 36 indicators of 
successful writing programs were identified. These indicators were subsequently tested for 
significance by comparison to seven continuously declining schools, 35 of the 36 indicators were 
found to be significant (AEL, 1997b; Coe, et al., 1999a). In sum, tlirough site visits to many 
Kentucky schools, more than 1 00 teachers, 200 students, and 50 administrators were interviewed to 
establish the 36 indicators (AEL, 1997b). These indicators encompassed issues such as 
administrative support, writing program coordination, and family and community involvement, in 
addition to the expected salient issues such as instructional strategies and professional development 
(see Appendix A for the 36 indicators and differences between the continuously improving and 
continuously declining schools). 

The strong emergence of a group of indicators related to teacher participation in professional 
development confirmed the wisdom of having modified an original objective to create new 
professional development programs immediately. In testing these indicators between improving and 
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declining writing programs, it was found that schools in which teachers had participated in 
professional development opportunities already available through KDE were more likely to evidence 
improved student writing than schools with untrained teachers. Thus the more salient need related 
to these indicators appeared to be heightening teacher participation in existing professional 
development, not revamping the programs themselves. 

School-Level Needs Assessment Instrument 

The second activity to emerge out of the modified objectives of this project was the 
development of a comprehensive needs assessment instrument, which was based on the indicators 
identified in the first activity. The needs assessment instrument originated from the collaborative 
research team’s activities in the process of identifying the indicators. It was very much an iterative 
design which grew out of using and revising the interview protocols in the series of site visits and 
writing the reports of findings from each school. Based on these activities, the research team (1) 
confirmed that talking to students was indispensable in evaluating a school’s writing program and 
(2) decided that writing the reports would be a valuable activity for stimulating schools’ self- 
awareness. Thus the team envisioned an instrument whereby schools, led by an in-house faculty- 
based Steering Committee, could assess for themselves the strengths and weaknesses of their writing 
program relative to the indicators, the key operational elements being input from all school 
stakeholders and self- (versus external) evaluation. Thus was conceived and developed a 
needs assessment for schools called the School Study of Writing Instruction (AEL, 1997a; AEL & 
KDE, 1998). Henceforth, it shall be referred to as the School Study, self-study, study, manual, or 
instrument. In accordance with the intent of KDE and AEL decision makers, the collaborative 
research team had designed an insightful approach for eliciting the voices of all school stakeholders, 
particularly those of students. The School Study is the instrument under consideration in this paper. 

The purpose of this paper is to describe the pilottesting and the fieldtesting of the School 
Study of Writing Instruction. Given their duration and complexity, each test of the instrument will 
comprise a separate section. 
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THE PILOT TEST 



Methodology 

The pilot test methodology is divided into two parts. The School Study was tested in a setting 
of use in its entirety (Method I, the full model), and sections of the instrument were either pilot tested 
or reviewed by other individuals (Method II, the sectional model). First, the data sources used for 
the pilot test are listed. 



Data Sources 

Eight data sources were used to complete the pilot test as described below. 

Steering Committee Feedback Form (data source for Method I only). The facilitator at the 
pilot-test school presented a brief form of 10 questions to the faculty-based Steering 
Committee upon the school’s completion of its study. Questions asked about time estimates 
in completing their tasks, the nature of their planning, their perceived need for facilitator 
assistance, and the utility of the study. The Steering Committee submitted their responses 
collectively on one form. 

Principal Feedback Form (Method I only). The facilitator presented a form of four 
questions to the principal at the conclusion of the study. All questions centered on the utility 
of the study. 

Facilitator Log (Method I only). The facilitator kept a log of how the pilot-test school 
carried out the steps specified in the School Study manual. 

Debriefing Meeting of the Collaborative Research Team and Typescript (Method I 
only). Less than a month after the completion of the pilot test, the school facilitator 
elaborated on the school’s study process and her role as observer/facilitator for the research 
team. The meeting was taped and transcribed. 

Interviews with the Facilitator (Method I only). Four informal phone interviews were held 
with the facilitator to glean additional detail about how the school carried out the study. 

Reactions Form (data source for Method II only). Participants who tested sections of the 
instrument were mailed this form along with their testing package. It consists of five 
questions, three of them in two parts, which solicit opinions about the clarity of manual 
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instructions, time estimates to complete the steps, the utility of the study for middle and high 
schools in particular, and circumstances that would propel middle and high school faculties 
to undertake the self-study. 

AEL Meetings and Notes (data sources for both Methods 1 and H). At five meetings of the 
AEL contingent of the research team over the course of two and a half months, background 
information about the Kentucky writing project and details about the pilot test was gathered. 
Notes of these meetings were consulted for writing the pilot test report. 

Cassette Tape (both Methods 1 and 11). A member of the collaborative research team 
recorded highlights of the pilot test background and findings for this study. 



Method 1: Participants 

An elementary school with 244 students in grades K-6 and approximately 25 teachers was 
chosen for pilot testing the entire instrument (Quality Education Data [QED], 1998). The school is 
located in an urban fringe of a mid-size city of western Kentucky (Burczyk, 1998). In this school’s 
district, 12% of the student population qualify for Title I funds, 90% are white and 9% black (QED, 
1998). The faculty accepted the invitation to participate in a full faculty meeting at which the 
principal briefly outlined the study and its purpose, and assured them of confidentiality. The 
principal was new to the school the 1998-99 school year. 



Materials 

The School Study of Writing Instruction (1998) is new and unestablished in the field. The 
study is a process divided into four main steps: (1) the interviews, (2) the report, (3) the ratings, and 
(4) the priorities. The interview questions, report topics, and rating categories correspond to the 36 
quality indicators of improving writing portfolio scores. They deal with issues of administrative 
support, professional development, school climate, writing program coordination, focus on writing 
portfolios and writing in general, instructional strategies, family and community involvement, and 
assessment of the writing program’s value for students. The interviews are the data upon which all 
subsequent analyses on the indicators are based. They are the linchpin because they come from four 
different role groups (district administrator, principal, teachers, and students), thus serving as a 
medium for closely comparing different parties’ experiences and perceptions of the writing program. 

The School Study is in manual format in a three-ring binder. It is comprised of an 
introductory overview and letter to the Steering Committee, as well as a section for each of the four 
main steps listed above. Also, all the forms for use in each step are compiled on computer disk. The 
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overview includes a “Time and Task Summary” that allots an estimated completion time for each 
of the activities involved in the four steps (see Appendix B). The confidentiality and in-house use 
of the information gathered are emphasized several times throughout the manual. 



Procedure 

One copy of the manual was mailed to the pilot test school, along with a version on disk in 
WordPerfect 8.0, in early November 1998. As an incentive to pilot test the instrument, a stipend of 
$1,500 was provided by KDE to the school. 

As the collaborative research team prepared for the pilot test, a critical question concerned 
how much facilitator assistance to provide to the school as it proceeded through the self-study 
process. It was conjectured that assistance would be most crucial for training the interviewer(s) and 
writing the report. Initially, the team planned to test six schools, two at each of three conditions of 
facilitation level: stand alone (i.e., no facilitation), local facilitation, and regional facilitation (AEL, 
1 998b). However, since a more expansive field test was already planned, in the interests of time and 
feasibility, the team decided to test just one school with an observer assigned to them who would 
monitor their progress and give suggestions only minimally. A retired Regional Writing Consultant 
(and “ex officio” member of the collaborative research team) was hired to serve as an observer of 
the pilot-test school. However, her role eventually became one more closely approximating an 
observet/facilitator in the process of the pilot test. 

Once the school had completed the study, the Steering Committee and principal gave 
feedback to the project team. Also, they gave permission for their findings to be reviewed by the 
team so that a qualitative assessment of the validity and reliability of the instrument could be made. 

A debriefing meeting among the Kentucky writing project team was held afterwards for the 
dual purpose of discussing the outcomes of the pilot test and preparing vwiting consultants for their 
upcoming facilitator/contact person role in the field test. At this meeting, and by phone and notes 
thereafter, the pilot-test facilitator described the pilot school’s experience in-depth. 



Method II: Participants 

Sections of the School Study of Writing Instruction were tested by individuals representing 
various entities in lieu of pilot testing the entire instrument at additional schools. The main impetus 
for this method was to gather some understanding of how well the instrument would work in schools 
at the middle and high school levels. 

Seven educators from four Kentucky school regions were asked to participate: five followed 
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through with their task, while two agreed to participate, but did not complete their task. One 
respondent also enlisted her spouse to pilot her assigned section, bringing the total number of 
participants to six. Of these, one was a regional social studies consultant, one a regional language 
arts consultant, one a high school teacher, one a middle school teacher, and two were current regional 
writing consultants (and new members of the collaborative research team) who recently had been 
either a middle or high school teacher. Finally, insofar as these “primary” subjects engaged the 
participation of others in their pilot testing, there were additional uncounted “secondary” subjects. 



Materials 

The sections of the instrument either reviewed or pilot tested in Method II were as follows: 
the guidelines for teacher and student interviewee selection and preparing interviewers (part of step 
1, the interviews), the teacher interview/survey (also step 1), and the student interview (also step 1); 
the report writing instructions (part of step 2, the report); and setting priorities (step 4, part of the 
priorities). No part of the ratings section (step 3) was reviewed because it had been given to one of 
the nonrespondents. A detailed description of the steps of the instrument is provided under the 
Materials subheading in the Method I section of this paper. 



Procedure 

Depending on the step involved, participants were asked to either review or pilot test their 
section. Each subject was supplied with the following: an introductory letter explaining the 
rationale for the School Study of Writing Instruction and the questions to be answered in the pilot 
test, an article describing the background work to the School Study (i.e., the establishment of 
indicators for improving writing programs), overviejv materials from the manual, instructions for 
pilot testing their designated section of the manual, their designated manual section, and a reaction 
form. Those participants piloting aspects of the interviewing process engaged other faculty and 
students in the task. Also, these participants were told to complete just part of an interview and then 
estimate the completion time. A nominal stipend of $50 was provided to the six participants upon 
receipt of their completed reaction form. 

The reaction form posed five questions, three of them in two parts. The questions asked 
whether 

1 . directions were clear and complete 

2. time estimates were realistic 

3 . a. circumstances existed in middle or high schools that would prevent them from 

undertaking the study 
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b. availability of outside assistance would be critical in the decision of middle or high 
schools to undertake the study 

4. a. schools would feel a need to do such a study 

b. (what) would be the impetus for schools to do the study 

5. a. the manual was “user-friendly” for middle and high school faculty 
b. any improvements could be made 

Findings 

The self-study was successful for the pilot-test school in Method I. One observed outcome 
was the Steering Committee’s practice of customizing the instrument while conducting the study, 
a noteworthy example of Berman and McLaughlin’s (1975) concept of “mutual adaptation” in 
education innovation implementation (Hord, 1987). The facilitator attributed the school’s success 
to a faculty sense of ownership of the program generated by faculty participation at the rating 
meeting. Moreover, the study process was efficient for the general faculty body; however, the 
compromise was that the process was time-intensive for the Steering Committee, as Committee 
members had been thorough in their planning and preparation. Yet overall, the process was 
parsimonious in that all steps up to, but not including, step 4 (the priorities) were accomplished in 
five weeks. 

The observer/facilitator’s assistance was judged to be critical, but not constantly needed. In 
her account, her best service was pointing out the essential pages of the study manual to the Steering 
Committee at their initial planning meeting, thereby making the study process seem manageable and 
straightforward. It is recommended that schools undertaking the School Study in the future have 
access to some level of assistance, whether it is provided by in-house or external facilitators. 
Findings in Method II also were informative, though not as comprehensive as those in Method I. 
Overall, participants found the manual to be exceptionally clear, well-organized, and user-friendly; 
nevertheless, it needs to be presented appropriately to faculty to minimize concerns based upon its 
size. The expected time commitment to conduct this study was the main concern of participants; 
nevertheless, participants felt that with facile presentation of the manual at the outset, the study 
might be willingly undertaken by middle and high school faculty. 



Conclusions and Recommendations 

Overall, positive conclusions are drawn about the self-study process in the pilot test. The 
faculty in the full model demonstrated that it was valuable to them, mainly due to their having 
assumed ownership of the steps in the process. The time commitment required to undertake the 
study was an issue of concern, although this was mitigated somewhat by the school’s ability to adapt 
the instrument to suit their needs. The facilitator’s role was deemed crucial at critical junctures; yet 
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creative recommendations for decreasing reliance on external facilitators are proposed. Validity and 
reliability are considered in interaction with perceived value of the study. 



Value 



In conclusion, the School Study of Writing Instruction has high and immediate utility for 
schools. With the possible exception of step 2, writing the report, it is a palatable, straightforward 
process for participants. Especially with a self-study, it is promising that a program of the school 
encompassing a large number of factors can be measured rigorously in a short amount of time; thus 
it appears to be a comparatively parsimonious process. Also, the faculty reported that they felt the 
study was valuable and expressed an appreciation for its data-based approach. Nevertheless, the 
Steering Committee members in Method I and the participants in Method II expressed concern about 
the time involved to complete the study. 

The facilitator attributed the study’s success to two manifestations of the faculty’s sense of 
ownership of the study. First, the Steering Committee assumed ownership of the tasks involved, 
particularly writing the report and leading the rating session. They immersed themselves in the 
process and were self-guided much of the time. That they chose to conduct the principal interview 
themselves, handled a considerable amount of planning, wrote the report unaided, and planned 
extensively before the rating session demonstrates their interest in the study’s outcomes. 

Second, the faculty-at-large also assumed ownership of the process principally by means of 
extensive discussion. The facilitator declared that the opportunities for discussion were by far the 
most valuable aspect of the study — both for gamering initial buy-in and giving teachers a stake in 
analysis and planning in the ratings session. Both the small- and large-group forums in the rating 
session contributed to session outcomes. What was salient about the discussions was that they drew 
full participation and were productively targeted. An intriguing note on the tenor of these sessions 
is that faculty examined indicators at which they were high-perfomiing as well as low-performing; 
so the study may be useful for illustrating strengths as well as weaknesses of a writing program, for 
allowing staff to take some pride in there strengths, even if no actions are to be taken in regards to 
them. Another factor that seems to have cemented the sense of ownership felt by faculty is that they 
were neither forced to do the study nor accountable to an outside authority for its outcome. Faculty 
were regularly reminded, including at the start of the rating session, that their analyses were 
exclusively for in-school use, which the facilitator asserts was important to them. In sum, as the 
research team had suspected early in the project’s development, the “self’ aspect of the study was 
a cardinal asset to this needs assessment. 

A third factor likely affected the actual and perceived value of the study. This was the data- 
based approach. The faculty liked seeing tangible results displayed in a tangible product such as the 
school profile. One faculty member commented that it validated the worth of their perceptions. 
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imbuing them with the stroke of authenticity. Also, it lent organization and cohesion to the plarming 
process, making obstacles seem solvable because they were being solved collectively. 

Recommendation 1. Incorporate guidelines in the manual that describe the importance of 
providing an adequate rationale for conducting the School Study to the faculty, i.e., the first step in 
“marketing” the process. Also include guidelines for the next step: clearly and concisely presenting 
the School Study manual to the Steering Committee. 

Recommendation 2. Include a suggestion that schools undertake the study in the spring, 
which is closer to the time the school plan is written for the following school year. 

Recommendation 3. Include guidelines in the manual that suggest at least two planning 
meetings and specify a general outline for them. Furthermore, the time frame should perhaps be 
lengthened to “4-6 weeks through step 3" and a separate time frame proposed just for step 4. 

Recommendation 4. Include guidelines that encourage administrators to consider ways of 
supporting Steering Committee members with release time and/or stipends to conduct the study (the 
stipend of $1,500 provided by the Kentucky Department of Education was intended only for schools 
which participated in the pilot and field tests). 

Recommendation 5. Revise guidelines to address the possibility of hiring interviewers and 
further emphasize the benefit of hiring typists. Schools have to ensure that interviewers are not well- 
known to the teacher and student interviewees, which most likely requires looking outside the school 
for them. In the pilot-test school, the interviewer chosen was an ideal choice because (1) she was 
already a paid staff member who could easily complete her task during working hours since she did 
not have a regular classroom and (2) she was new and thus relatively unknown to the teachers and 
students. These propitious circumstances may be difficult to replicate. In regards to typing, the 
Steering Committee Chair had typed both the interviews and report, even though the manual 
suggestion is to consider hiring others, such as paraprofessional staff. In hindsight, she suggested 
that hiring someone for these tasks would have been a better option. Of course, there is the 
possibility that some individuals may be willing to serve as interviewer or typist on a volunteer basis. 

Value Added by Customization 

The school added value to the study by customizing the School Study instrument protocol to 
suit its purposes for completing the study. That the school was able to customize it easily but still 
remain within the confines of the study’s purpose is advantageous for the survival of the instrument 
in future schools — it is malleable yet powerful. Ideally, the protocol would be flexible because it 
will be new and perhaps foreign to each school that undertakes it; if the study were regimented such 
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that it had to be conducted exactly as written, it would be more difficult to implement widely. In 
their research on educational innovation, Berman and McLaughlin (1975) used the term mutual 
adaptation to describe the optimum pattern for implementing a new innovation (Hord, 1 987, p. 1 4). 
In mutual adaptation, an innovation and the school setting are adjusted to fit each other. 
Furthermore, unlike other patterns of innovation implementation that target the student and ignore 
the teacher, mutual adaptation works because it recognizes the teacher as “an irreducible middle 
component in implementation” (Hord, 1987, p. 15). This understanding reinforces the importance 
of teachers’ sense of ownership in planning and analyzing the writing program. 

Table 1 displays the pilot school’s 33 adaptations ofXhQ School Study of Writing Instruction. 
A review of the adaptations in Table 1 in the Results section reveals that many revolve around time 
and efficiency, although some are clearly more related to quality control of the study. In all, these 
“shortcuts” made the progression between steps faster and smoother, yet they required that the 
Steering Committee invest more time in planning and preparations. Also, a few were either neutral 
or perhaps counterproductive. 

Recommendation 6. Adopt adaptations listed in Table 1 as deemed useful and incorporate 
them into the School Study manual. 

Role of the Observer/Facilitator 

Although initially expected only to observe the pilot test, the observer/facilitator emerged 
predominantly as a facilitator, which ultimately contributed a great deal of additional knowledge to 
pilot-test findings. The two quintessential roles assumed by the facilitator were initiator of the study 
process and director at subsequent critical junctures. The research team predicted that the facilitator 
would be most needed for training interviewers and leading the report writing. In fact, she spent only 
minimal time training interviewers and none at all doing the latter. These findings undergird the 
viability of schools conducting the study in the future with little or no assistance. 

Nonetheless, although the Steering Committee assumed responsibility for a great deal of the 
planning and work, it is evident that they relied on the facilitator’s direction at critical junctures. 
Also, they expressed doubt in their ability to have completed the study without her. Furthermore, 
as the facilitator’s role extended only through the ratings session and some preliminary priority- 
setting, it is not known how successful the priorities-finalization and actions-planning activities were 
for the School Consolidated Planning Committee. Perhaps facilitator guidance would be useful for 
these tasks as well. 
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Table 1 : Adaptations of the School Study of Writing Instruction by Pilot School 

Initial Planning: 



14 



* 1 . (Principal) Made a copy of the manual for her own reference 

2. (Principal) Garnered initial buy-in at a faculty meeting 

3. (Faculty) Nominated only teachers for Steering Committee 

4. (Steering Committee) Held two planning meetings 

*5. At the first planning meeting, (the facilitator) “broke down” the manual for the Steering Committee in order 
to demonstrate the manageability of the process 

6. Prior to the second planning meeting, (Steering Committee) read the manual and completed interviews 

7. At the second meeting, read the rating guide and discussed the ratings session procedures 

8. Pre-assigned small groups for the ratings session and gave them option of meeting prior to the session 

9. Pre-appointed themselves (i.e.. Steering Committee members) as small- group leaders 

10. Set goal and back up plan for rating session 

1 1 . Allotted entire workday for rating session 

12. Planned what types of indicators to target for improvement (the priorities) 

Interviews: 

13. Chose one interviewer versus several so as to minimize interviewer training time 

14. Dismissed the possibility of using higher grade level students as interviewers 

15. Chose option of Steering Committee member (rather than interviewer) to interview the principal 

16. Chose option of interviewer (rather than Steering Committee member) to interview district administrator 

17. Modified the random selection process for teacher interviewees 

*18. (Interviewer) Interviewed teachers verbally only, dismissing the written survey option 
19. Selected student interviewees on the day of their interviews 
*20. Interviewed students in pairs only 

21 . Alternated writing and recording tasks differently than stated in manual (applicable only to 
practice interview in which there were two interviewers) 

22. (Interviewer) Relied on tapes more than notes to write up the interviews 

*23. (Steering Committee Chair) Condensed teacher and student interviews onto one form each 

Report: 

24. (Steering Committee Chair) Acted as a resource person for the other Committee members 
writing the report 

25. (Steering Committee) Did assigned report sections in pairs 

26. Did not use the “Other” spaces provided for recording information not elicited in interviews 
Ratings: 

27. (Steering Committee) At ratings session, immediately directed faculty to small groups 
*28. Separated rating guide into sections for small groups 

29. Within each group, read rating guide examples aloud 

30. Transferred all ratings onto school profile on overhead, instead of verbally sharing results 

31. Once the school profile was displayed, opened up discussion of ratings in the 
large group for further consensus-building 

32. At conclusion of ratings session, engaged in informal priorities-setting 

Priorities: 



33. Assigned completion of priorities-setting to the SCP Committee, instead of to the full faculty 
*Note: These adaptations were suggested in time for the field test. 
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In sum, the degree of need for facilitation remains unclear for future implementation. The 
pilot-test facilitator felt that a facilitator should be geographically close to the school to be available 
fairly constantly, but not necessarily present during the entire process. It is not feasible for the 
Kentucky Regional Writing Consultants to continue to act as facilitators beyond the pilot and field 
tests because there are only eight statewide. 

Recommendation 7. To further elucidate the need for a facilitator, the field test is currently 
being employed at two conditions of facilitation — half with a facilitator and half with just a phone 
contact person. The collaborative research team hopes to discover how well the latter, independent 
group of schools manages without a full-fledged facilitator. 

Recommendation 8. As it is predicted that schools will either want or require some degree 
of assistance, it is recommended that external facilitators be identified, trained, and made available 
to desiring schools in the fall of 1999 as it is fiscally viable. 

Recommendation 9. Another way of providing facilitator assistance may also be explored. 
The pilot-school believed that the possibility of effective in-school facilitators is feasible. She 
expressed that if one faculty member is thoroughly trained in the School Study process, a school can 
carry it out with little to no assistance. That the Steering Committee Chair in the pilot-test school 
was nominated as a quasi-facilitator for a part of the process, and is a member of the School 
Consolidated Plarming Committee that will plan the new actions, undergirds the possibility of strong 
teacher leaders acting as facilitators. In-school facilitators would need to be capable of engineering 
the crucial steps, such as effectively outlining the manual for the Steering Committee, training 
interviewers, and guiding the report- writing and rating session. The facilitator’ s proposal for training 
teacher leaders is to hold a region-wide meeting for Steering Committee members from various 
schools. 

Recommendation 10. The pilot-school facilitator also proposed conducting the rating 
sessions of several schools contemporaneously at a central site, led by trained in-house facilitators 
and overseen by district, regional, or state staff. Perhaps schools could undertake the earlier and later 
steps in the process with only phone contact from outside staff. 

Recommendation 11. The central issue for consideration in deploying facilitators is cost. 
The amount and type of facilitator assistance offered to schools will determine a large portion of the 
cost of implementing the self-study. Schools are permitted to use monies for consolidated planning 
to implement such a program; helping schools identify these funds is a task for the regions and 
KDE. Offering regional and state assistance in larger forums such as the region-wide m.eetings 
discussed previously will reduce the cost. Yet doing so may also transfer cost to schools as they 
support training of teachers as facilitators. 
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Recommendation 12. The research team has considered separating the manual into two 
volumes — one containing the instructions and the other just the forms. 

As indicated earlier, the presentation of the manual absent the guidance of an outside 
facilitator became a central issue for consideration when the research team perceived how valuable 
the facilitator’s presentation was for the pilot-school Steering Committee in Method I. The concern 
is that the size of the manual may immediately alienate faculty, either tempering enthusiasm for 
undertaking the study or diminishing the quality of study outcomes. It is noteworthy that the 
participants in this study who did not have the benefit of a facilitator, those in Method II, 
unanimously raised a concern about the magnitude of the study, based on the manual’s size. One 
did qualify her concern by stating that schools’ willingness would depend on quality presentation 
of the manual. Whereas participants who did have the benefit of a facilitator. Steering Committee 
members in Method I, appraised their facilitator’s initial presentation of the manual as the greatest 
benefit of her assistance, and doubted their ability to conduct the study without her. Thus for schools 
that might not have facilitators in the future, it is necessary to make the School Study appealing to 
them as a stand-alone product. If securing and deploying facilitators as described in 
recommendations 8 and 9 proves to be difficult for schools in the future, modifying the manual is 
a tangible way in which the threat of its large appearance may be minimized. 

The preceding discussion has centered on the need for a facilitator. Yet a participant in 
Method II offered an interesting perspective on the desire of schools to have a facilitator. In her 
estimation, the presence of a facilitator would infringe on the confidentiality of the study and thus 
possibly mitigate a school’s willingness to undertake it. The research team believes that schools will 
value the confidentiality of this study, not out of a desire to keep results secret from individuals at 
random, but based on the reassurance that their results are not going to be “measured” by an outside 
entity to which they are frequently accountable. The facilitator asserted that this aspect of 
confidentiality was indeed important to the pilot-test school. In sum, an individual facilitator would 
probably not pose a threat to a school’s sense of confidentiality. Yet the above participant’s 
perspective is useful nonetheless, reinforcing the importance of faculty ownership in school 
programs. It underscores the fact that facilitators should vigilantly refrain from controlling or 
criticizing the study process of a school, which might undermine a faculty’s sense of ownership. 
Instead, they should consider themselves “helpful guests.” 



Validity and Reliability 

Validity and reliability are requisite issues for consideration whenever a new instrument is 
tested (Huck, Cormier, & Bounds, 1974). In the course of performing a reliability test, an internal 
validity threat was identified in relation to the “Table of report sections and data” significant enough 
to warrant revision. Other minor problems in various components of the instrument were identified 
as well. These weaknesses concerned accurate data transposition tlirough the pipeline from the 
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interviews to the report, ratings form, and school profile. 

To correct one of these threats to validity, the “Table of report sections and data” was 
reconfigured after the pilot test to more specifically link each report topic with its corresponding 
interview questions (or data reference items). Three members of the collaborative research team 
undertook the task individually and then compared their tables. However, each generated a different 
table, thus presenting a new difficulty. The issue was resolved for the field test by removing the 
original table altogether and choosing not to replace it with one of the new tables. The arguments 
for removing the table were that (1) responses to interview questions do not bear a linear relationship 
to specific report topics (hence the research team’s difficulty trying to create a uniform table among 
themselves) and (2) the desired global focus of the report could be better attained by leaving open 
the field of possible data reference items for the writers to choose from. 

Recommendation 13. Yet despite the difficulty of creating a new “Table of report sections 
and data” it is recommended that one of the three new tables be inserted in the final version of the 
School Study of Writing Instruction. The research team will await the outcome of the field-test 
schools’ experience before deciding whether to eliminate or include it. This study requests teachers 
with limited time to draw conclusions from concrete data instead of their conventional wisdom. 
Therefore, every step should be as clear and “mechanized” as possible so that the process feels fluid 
to the participants. Shuffling through numerous interviews with no specific direction may overload 
report writers, paradoxically turmeling them into a narrow focus in which they can only manage to 
skim conspicuous aspects of the interviews. Therefore, the argument for including a new “Table of 
report sections and data” reflects the original intent for having a table — to help report writers identify 
relevant data reference items so that they write complete, multifaceted report topic responses. 

It is true that the interview questions (data reference items) are not neatly discrete or 
exclusive in their relevance to different report topics. However, further reflection on one of the 
newer tables reveals that data items do bear a discernible linear relationship to the topics — it just 
happens to be a multilinear one. That is, data generated by one interview question may relate to 
several report topics. Therefore, it is possible to specify the data reference items. Since it is 
possible, it may also be preferable: a report writer’s decision to ignore cued data is probably easier 
to make than the decision to include data that are not cued and have to be hunted. In sum, specifying 
as many data items as possible beforehand for report writers will likely enhance their efficiency by 
minimizing data-hunting time, and probably help them create a more global representation of their 
school’s writing program than if they had no table. 

Recommendation 14. If the new table is adopted in the final instrument, written and verbal 
guidelines for writing the report should be modified accordingly. The brevity of the pilot school’s 
report was perhaps as much owing to the instructions given in the manual and by the facilitator as 
it was to the vagueness of the original table. The recommendations given in the manual are to record 
the “gist” of report topics using the “greatest brevity.” In retrospect, these recommendations are 
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more fitting for note-taking while reading the interviews than for report writing. 

Ideally, instructions would provide insight about how to incorporate detail meaningfully, 
while keeping the report topic responses relatively brief (i.e., short paragraphs). 

Finally, an additional two hours should be added to the original estimated time for report 
writing (1-2 hours per person). Although the teachers estimated that this task took them 2-3 hours 
individually, they made general statements that it felt like quite a bit of time, leading the facilitator 
to think that accurate estimates were probably larger than that. (The change should be made in the 
“Time and Task Summary” in the overview as well as in the text of manual guidelines.) 

Recommendation 15. Thus far, adaptations to the School Study of Writing Instruction have 
been considered in Table 1, and revisions to parts of the instrument such as the “Table of report 
sections and data” have been considered. These adaptations and revisions are briefly noted again, 
along with other proposed revisions to the instrument, in Table 2. It is recommended that all those 
deemed useful be adopted for revision to the School Study of Writing Instruction manual and a final 
version of the manual be produced (AEL, 1997a). 

Recommendation 16. The School Study of Writing Instruction is recommended for ongoing 
implementation in Kentucky schools. The field test currently is exploring schools’ need for 
facilitator assistance in more depth. 

Recommendation 17. The School Study of Writing Instruction should be developed as a 
stand-along product, not only for Kentucky writing programs, but for writing programs nationwide. 
An effective marketing plan should be written to meet this end. The manual should be produced so 
that it may stand alone in consideration of the fact that some schools that purchase it may not have 
facilitator assistance. 

Recommendation 18. The research into indicators of successful writing programs as the 
basis for the School Study of Writing Instruction may be considered a model for development of 
other curricular needs assessment instruments (AEL, 1997b; Coe et al., 1999b). Not all of this 
research base is specific to writing instruction; some parts are generic to teaching, learning, and 
school efficacy. Thus the School Study may be easily adapted as a self-study needs assessment for 
content areas other than writing. This being so, such a product with either generic topical 
applicability, or several products each tailored content-specific, should be produced and also 

1997a). A separate marketing plan should also be written to meet 

Revisions to the School Study of Writing Instruction 



disseminated nationwide (AEL, 






Table 2: Proposed 
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Overview: 


1. Perhaps change time frame to longer than one month 

2. Recommend that schools start the study in the spring 

3. (In the Time and Task Summary) Allot more total time for the interviewing step, including time to type 
compilations of the teacher and student interviews 

4. (In the Time and Task Summary) Allot more time per individual for the report step, including time to type 
the report 

5. (In the Time and Task Summary) Allot time for two Steering Committee initial planning meetings 


Initial Planning: 


6. Incorporate a section of guidelines in the manual for conducting two initial planning meetings; 
specify the content to cover at each meeting and, what to read in preparation for the second one. 

7. Adopt all adaptations in Table 1 as considered helpful and appropriate 


Interviews: 


8. Adopt all adaptations in Table 1 as considered helpful and appropriate 

9. [Make corrections to weaknesses listed in Table 2, especially the errors in numbering sequence.] 
* 10. If feasible, select two interviewers instead of one, who then conduct each interview together 
*11. Interview students in groups no larger than pairs 


Report: 


12. [Adopt all adaptations in Table 1 as considered helpful and appropriate.] 

13. [Make corrections to weaknesses listed in Table 2, especially to the “Table of report sections and data” and 
the report writing instructions.] 

* 14. (If a new “Table of report sections and data” cannot be created;) remove the table. 

15. Remove instructions to sort interview forms before writing the report (this activity having been made 
unnecessary by the creation of a “composite” form for the teacher and student interviews). 


Ratings: 


16. [Adopt all adaptations in Table 1 as considered helpful and appropriate.] 

17. [Make corrections to weaknesses listed in Table 2, especially the omission of indicators in the rating guide, 
rating form, and school profile.] 


Priorities (and Actions): 


18. [Adopt all adaptations in Table 1 as considered helpful and appropriate.] 

* 19. Provide more specific guidance in the manual about incorporating new actions into the School 
Consolidated Plan, such as a guide (Impact Check) and sample (Action Plan) 


Other Minor Revisions: 


20. On the report form, letters under report topics 1 A and IB should be capitalized 

21 . In the ratings section, change the wording of “small- group facilitator” to “small- group leader.” 

22. Also in the ratings section, fit the computer-generated sample school profiles on one page like the school 
profile to be used by the school, allowing for better visual comparison. 



*Note: These changes were suggested in time for the field test. 



THE FIELD TEST 
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Methodology 

The field test methodology is divided into two conditions. In the facilitated condition, five 
schools were assigned a facilitator from the collaborative research team to lead them in the School 
Study. In the independent condition, six schools were assigned a contact person from the team to 
be available for telephone consultation. The data sources, participants, materials, and procedure are 
described next. 



Data Sources 

The data sources used to complete the field test are described below. 

Steering Committee Member Feedback form. A form of 1 1 questions accompanied the 
mailing of the field-test materials. The research team requested that it be distributed to each 
participant on the Steering Committee upon completion of the study. Questions related to 
time estimates for completing the study, the nature of planning activities, perceived need for 
facilitator assistance, and the utility of the study. The facilitator assistance question was 
customized to each of the two conditions of schools. Forms were returned to the research 
team between late May and late June 1999. 

Principal’s Feedback form. A brief form of four questions also accompanied the original 
materials with the request that the principal complete it at the conclusion of the study. All 
questions centered on the utility of the study. Forms were returned to the research team 
between late May and July 1999. 

Phone interviews with Steering Committee chairs. The Steering Committee chair at each 
field- test school was contacted by a member of the collaborative research team and asked 
several questions regarding the performance of the specific steps of the School Study. 
Because a few schools were not finished with the study by the date of the first phone call, 
some follow-up calls were necessary. The phone interviews were conducted between mid- 
May and mid- July 1999. 

Facilitator logs. Facilitators each kept a log of interactions with their assigned school(s) to 
track their progress in carrying out the School Study. Some contact persons (for the schools 
in Condition 2) kept a log as well, although they were not asked to do so. These were 
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Roundtable debriefing meetings of the collaborative research team. At two meetings in 
May and July 1999, near the completion of the field test and at its conclusion, facilitators. 
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contact persons, and other members of the team shared their insights about schools’ progress. 

Phone and e-mail contacts with facilitators and contact persons. Members of the 
collaborative research team who did not serve as a facilitator or contact person 
communicated with those who did to check schools’ progress and gather data. 



Participating Schools 

A pool of 1 2 schools was developed by nominations of the collaborative research team. The 
field-test nominations were purposefully selected from the eight education regions of Kentucky to 
represent different enrollment sizes, grade levels, and geographic and demographic regions (see 
Table 1 ). The research team was interested in discovering the School Study ’s utility in various school 
contexts because writing portfolio requirements apply to all schools. The Kentucky Commissioner 
of Education sent an invitation letter to the nominated schools, which included an offer of a $1,500 
stipend, to which 1 1 schools responded. 

Diverse representation is evident for most characteristics shown in Table 1 except locale type 
and race. (Location is more diverse than it appears given the geographic and education region 
demarcations of Kentucky.) Regarding locale type, most field-test schools came from small towns.' 
None came from the two largest locale types — a large city or an urban fringe of a large 
city — however, one was in a mid-size city. Regarding race, most schools came from a district with 
a majority white population. Although this sample is not extremely diverse in terms of locale type 
and race, it is believed to be to an extent that is representative of the state. 



Table 3: Characteristics of the Eleven Field-Test Schools 



'These are Johnson’s locale type definitions in abbreviated form (Burczyck, 1998): 

A rural area has a population of fewer than 2,500 inhabitants. 

A small town has a population of 2,500 to 24,999. 

A large town has a population of at least 25,000. 

An urban fringe of a midsize city is a place within the metropolitan area of a mid-size city. 

A midsize city has fewer than 400,000 inhabitants and a population density smaller than 6,000 people per square mile. 
An urban fringe of a large city lies within the metropolitan area of a large city. 

A large city has at least 400,000 inhabitants and a population density of at least 6,000 people per square mile. 
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Enrollment 


Grade Levels 


Location/KY I 


1 Locale Type" 


Race*’ 


Title L 


# schools 


1 # schools 


# schools 


# schools 


# schools 


# schools 


0-300 


PreK-6 


North orNW 


Rural area 


> 95% white 


1 - 10% 


3 


3 


2 


2 


8 


2 


301 - 600 


PreK-8 


East or SE 


Small town 


88%W; 10%B 


1 1 - 20% 


4 


3 


6 


6 


1 


3 


601 -900+ 


Middle 
(4-8; 7-8 gr.) 
2 


Central 

1 


Urban fringe of 
mid-size city 

2 


80%W; 18%B 

1 


21-30% 

2 


4 


High 

(7-12; 9-12 gr.) 
3 


Western 

2 


Mid-size city 

1 


60%W; 39%B 

1 


31 - 50% 

4 



Note. Read each column of data as a separate entity: there is no relationship between successive cells in a row. 



“Data are from Burczyk (1998). 

'’Data are from Quality Education Data (1998). Percentages apply to the district in which each school resides. 



Figure 1 shows the latter two characteristics of Table 1 — race and Title I eligibility — in 
relation to each other. 



Materials 

The School Study of Writing Instruction (AEL & KDE, 1 999) was revised based on the pilot 
test. The study is a process divided into four main steps: (1) the interviews, (2) the report, (3) the 
ratings, and (4) the priorities. The interview questions, report topics, and rating categories 
correspond to the 36 quality indicators of improving writing portfolio scores. They deal with issues 
of administrative support, professional development, school climate, writing program coordination, 
focus on writing portfolios and writing in general, instructional strategies, family and community 
involvement, and assessment of the writing program’s value for students. The interviews are the 
data upon which all subsequent analyses are based. They are the linchpin of the study; because they 
come from four different role groups (district administrator, principal, teachers, and students), they 
are a medium for closely comparing different parties’ experiences and perceptions of the writing 
program. 
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Figure 1: Depiction of Title I Eligibility by Racial Composition per School(s) 



8 schools 
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31% 
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88% white & 
10% black 



80% white & 
1 8% black 



60% white & 
39% black 



Racial Composition 

Note. Locale type data are from Burczyk (1998) and other data are from Quality Education Data (QED) (1998). 
Percentages reflecting race and Title 1 eligibility apply to the district in which each school resides. 



Content validity of the School Study instrument was determined in two stages, each 
consisting of several steps. The first stage involved identifying the variables for examining writing 
programs (Coe et ah, 1999a, 1999b). First, indicators were initially derived from the parameters of 
Kentucky’s writing program (see KDE, 1994a). Second, the collaborative research team developed 
draft interview protocols around the initial indicators, tested them in exploratory visits to seven 
Kentucky schools, and refined them. Third, the research team identified 43 schools with consistently 
improving writing scores and 1 8 with consistently declining scores. Based on site visits to 22 of the 
improving schools and 7 of the declining schools, in which more than 100 teachers, 200 randomly 
selected students, and 50 administrators were interviewed, team members established 36 indicators 
of successful writing programs (Coe et ah, 1 999a, 1 999b). A comparison of the difference in mean 
scores between the 22 schools with improving scores and 7 schools with declining scores verified 
the indicators’ capability to differentiate writing programs. These steps encapsulate the process by 
which the indicators were validated. 
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The second stage of establishing validity involved creating the School Study of Writing 
Instruction as a usable instrument for schools. First, it was designed around the indicators identified 
in the research stage. Second, it was designed in such a way to foster the involvement of several 
school stakeholder groups in studying a school’s writing program, particularly the inclusion of 
students as informants of writing instruction. Finally, to enhance utility, the School Study of Writing 
Instruction was revised through several iterations based on meetings of the collaborative research 
team, on the pilot test, and on this field test. 

The School Study handbook is a process manual in a three-ring binder. It includes an 
introductory overview and letter to the Steering Committee, as well as a section for each of the four 
main steps listed above. The forms for each step are compiled on computer disk. The overview 
includes a “Time and Task Summary” that provides estimated completion times for each of the 
activities in the four steps (see Appendix B). Confidentiality and the intention of in-house use of the 
information gathered are emphasized several times throughout the manual. 



Procedure 

Two copies of the School Study of Writing Instruction handbook were mailed to each of the 
1 1 schools that had agreed to participate, along with a version on disk in WordPerfect 8.0, in late 
January 1 999. The materials were accompanied by an introductory letter from the research team to 
the principal. As an incentive to field test the instrument, a stipend of $1 ,500 was provided by KDE 
to each school. Also, a letter detailing procedures was mailed to facilitators and contact persons. 

Six KDE writing consultants (members of the collaborative research team) were assigned 
among the 1 1 field-test schools to serve as either facilitator or contact person, each assigned to one 
or two schools in her region. Five schools had a facilitator, who would be regularly present and 
involved with guiding the study (Condition 1, facilitated. schools), and six schools had a contact 
person, who would be available for telephone consultation (Condition 2, independent schools). 
Facilitators and contact persons began calling their assigned schools in late January 1999 to set up 
meetings or to answer schools’ questions and check their progress regularly. During and at the 
conclusion of the field test, feedback was gathered from field-test participants via the several data 
sources described earlier. 

Each field-test school’ s experience of conducting the School Study was assessed on whether 
or not it produced some successful results. In this effort, successful results means that (1) a school 
completed the study to a logical interim point with adequate attention given to at least the first three 
steps, (2) a majority of principals and Steering Committee members found the study to be a 
worthwhile experience, and (3) principals and Steering Committee members anticipated benefits or 
changes to the writing program as a result of doing the study. The quality of the methods used by 
a school to carry out its study was not judged relative to successful results; however, the suitability 
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and quality of schools’ methods are discussed. Also, some schools experienced factors such as 
attrition by Steering Committee members and less than total faculty participation at the rating 
meeting; however, these factors were not counted against the success of these schools because doing 
so would negate the accomplislunents of the individuals who did participate. 

In addition, some schools gave permission for their findings to be reviewed by the research 
team so that a reliability assessment of the School Study handbook could be made. Specifically, the 
interviews and report were requested fi'om each school for use in assessing how efficaciously field- 
test schools transcribed interview data to the report. A sample of five of the possible 36 report topic 
responses was assessed, the same five per each report (topics 9A-E). Accuracy, breadth of data items 
considered, and level of detail were the main criteria for comparison. A point system was devised 
for measuring these criteria: as long as a response accurately included most relevant data items, it 
was considered entirely or approximately equivalent to the reviewer’ s comparison sample (and thus, 
representative of the data being measured); if a response accurately included 30-67% of data items, 
it was rated partially equivalent to the reviewer’s sample (i.e., partially representative); a response 
including fewer than 30% of the data items was considered barely equivalent (i.e., barely 
representative). 



Findings 

This section provides a synopsis of each field-test school’s experience with the School 
Study of Writing Instruction. 



Condition 1: Facilitated Schools 

The five schools described next were each assigned an external facilitator from the 
collaborative research team to guide them through the School Study process. 



Facilitated School #1 

FSl was motivated to conduct the School Study and achieved some successful results. 

The school spent nine weeks on the study, about twice the amount of time projected, and 
Steering Committee members each spent approximately 10 hours, reporting that interviewing and 
report writing took the most time. The facilitator observed that Steering Committee members 
prepared well for their meetings and tasks throughout the study process. Two members of the 
Steering Committee, the assistant principal and a parent, plus another parent not from the 
Steering Committee, conducted interviews; some teacher surveys were employed and students 
were interviewed mainly in groups. Also, FSl included individuals from its feeder school in its 
interview sample. The Steering Committee chose to write the report together in a meeting rather 
than work separately. For the rating meeting, the Committee developed a reflection form for 
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gathering faculty feedback about the study process. According to responses on this form, faculty 
learned that teachers and students in the school had different understandings of the writing 
program. Two Steering Committee members echoed this finding, saying that this realization was 
the most helpful aspect of the study process. Faculty also reported learning that assistance for 
and collaboration with content area teachers outside language arts were pressing needs, and 
identified several other needs as well. 



Facilitated School #2 

FS2 completed its study in five weeks. Overall, FS2 reported having a negative 
experience with the School Study. The principal did not participate, and the Steering Committee 
chair reluctantly accepted the position of chair. The Steering Committee chair ended up doing a 
substantial amount of the interviewing, report, and compilation work (40+ hours total). Neither 
did the encouragement of the Highly Skilled Educator (HSE) seem to fortify the Steering 
Committee’s motivation. Despite the availability of an external facilitator, the Steering 
Committee did not contact her for assistance beyond what services were offered. The purpose 
for doing the school study did not appear to be understood by the Steering Committee or faculty. 
The Steering Committee and faculty meetings began late and in a state of unpreparedness. 
Members of the Steering Committee reported that they would have liked a longer period of time 
(and more resources) in which to carry it out. Some teachers were interviewed via survey rather 
than via verbal interview. Various Steering Committee and faculty members’ assessments of the 
teacher and student interviews were that teachers’ responses were generally not representative, 
while students’ responses were. The majority of Steering Committee members felt that the 
school study was a waste of time. The principal’s feedback stated that the study was worth the 
effort, in marked contrast to the majority opinion of the Steering Committee. 



Facilitated School #3 

FS3 completed its school study in six weeks with some successful results. The study 
commenced at the prodding and initiative of the facilitator and in some confusion. The school 
principal had formed the Steering Committee without informing designated members of its 
purpose; however, once the Steering Committee understood and undertook its work, the study 
gained momentum and organization. Two drawbacks to the Steering Committee’s progress were 
that (1) the principal did not participate and (2) two Steering Committee members did most of 
the work (due to a combination of their own volition and perceived expectations of them). The 
Steering Committee never contacted the facilitator beyond what services were offered; the 
facilitator initiated all contacts and assistance. The facilitator was involved in some way at every 
step of the study process, including conducting a few interviews and facilitating the rating 
session. The Steering Committee was reluctant to engage interviewers from outside the school, 
so two of the members conducted the bulk of the interviews while the facilitator performed the 
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remainder. They found that interviewing students in groups of 2-3 worked well; however, it was 
sometimes necessary to rephrase questions so that students could understand them. Because the 
faculty in the rating session believed that the interviews and report did not accurately reflect their 
writing program, they had difficulty reaching consensus. The facilitator reported that, 
nevertheless, the discussion produced some shared understandings. 



Facilitated School #4 

FS4 completed its school study in five weeks. Although FS4 experienced a delay of 
several weeks in beginning the study, the school completed it with some successful results. The 
principal participated actively on the Steering Committee; however, the study began in some 
disarray. Not all individuals had been notified of their designation to the Steering Committee or 
informed of the study’s purpose before the first meeting. The Steering Committee was 
composed of 9 or 10 individuals, including at least one parent. The Steering Committee 
continued to exhibit some disorganization; no meeting was attended by all members and the 
Committee had to rush to finish by the end of the year given their delay in beginning the process. 
The facilitator was integral to starting the study on its course and active in several of the tasks 
involved. Several parents served as interviewers. One of the co-chairs reported that students 
sometimes had difficulty understanding interview questions. At the rating session, some faculty 
members expressed concern that parents perhaps were not the best choice as interviewers 
because they were not sufficiently knowledgeable of writing program jargon and, therefore, did 
not know when to probe students for more precise responses. Another obstacle in the 
interviewing task was that the district administrator refused to be interviewed, instead submitting 
only brief responses in writing. 

The faculty rating meeting far surpassed the Steering Committee’s expectations in scope. 
After rating, they progressed to discussion and reflections on next steps. The Steering 
Committee also held a meeting after the rating session to begin step 4 of the study process. Most 
Steering Committee members felt that the outcome of their self-study was worth the effort. 
Individuals’ time spent on the study ranged between 2-18 hours. They listed the rating meeting, 
interviews, and/or report as the most helpful elements of the School Study. Compiling the report 
and meetings were the activities that required the most time. 



Facilitated School #5 

FS5 completed its school study in seven weeks with some successful results. According 
to the facilitator, the assistant principal was a strong leader and assumed the position of Steering 
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Committee chair. The principal also participated, and the Steering Committee and faculty were 
eager to do the self-study. The Steering Committee regularly availed itself of the facilitator’s 
assistance even though Committee members believed that they could have conducted the study 
without such assistance. Nevertheless, they reported that the interviewing step would have been 
difficult to accomplish without additional help from other individuals, specifically finding the 
time needed to interview. Two substitute instructional assistants conducted the interviews as a 
pair. They interviewed students on an individual basis. Several participants in the study 
expressed the need for more teacher and student interviews for greater representativeness. To 
write the report, the Steering Committee worked together over a period of three days. The 
faculty-wide rating session was conducted as an entire group rather than in small group format. 

Three of the four Steering Committee participants assessed that the outcome of the 
School Study was worth the effort; one said somewhat. The time spent by individual members 
ranged from 6-24 hours, various tasks associated with the interviewing step taking the most time. 
Despite the time required to do the interviewing, they ranked the interviews as the most helpful 
aspect of the study. The Steering Committee planned to involve the entire faculty in setting 
priorities for the writing program, although they postponed the meeting planned for the end of 
the school year to a professional development session before the start of the following school 
year. The facilitator predicted success for FS5 in following through with the study given the 
Steering Committee’s promising expressions of intent and outlook. 



Condition 2: Independent Schools 

The six schools described next were each assigned a contact person from the collaborative 
research team to answer questions about the School Study process via phone or e-mail. The intention 
was for contact persons to be less involved with the study process than the facilitators would be with 
the facilitated schools’ process. 



Independent School #1 

ISl completed its school study in six weeks with some successful results. The principal 
chaired the Steering Committee and facilitated the study process. At the committee’s first meeting, 
he presented the School Study handbook to the committee. He called the contact person once with 
a few questions; she initiated other calls to monitor the school’s progress. The committee chose a 
parent to be the interviewer. The interviewers initially interviewed students on an individual basis, 
but switched to groups of 2-3, which they reported was preferable. The principal and two of the 
three Steering Committee members felt that the school study was worth the effort. The principal was 
appreciative of the opportunity to do it. Steering Committee members, aside from the principal, each 




34 



29 



spent between IV 2 to 12 hours on the study, report writing being the task that required the most time. 
As a result of the rating meeting, IS 1 planned to increase the participation of non-accountable grade 
teachers in the writing program. The principal expected that they would continue with the action 
plan in the next school year. One member expounded on a recommendation for revamping the 
format of the rating session, saying that it should have more closely emulated a workshop in which 
teachers modeled writing and left with a specific set of actions to implement or expect regarding the 
writing program. 



Independent School #2 

IS2 completed its school study in six weeks with some successful results. The principal was 
eager to do the study and reported being pleased throughout the process. The contact person initiated 
all calls to the school but one that came from the Steering Committee chair. Participants at IS2 had 
few questions. The Steering Committee was composed of five or six members and the writing 
cluster leader from the district office served as the chairperson. Due to the time constraints of others 
on the Committee, the chair organized the study and carried out most of the tasks herself The chair 
interviewed students in groups of four. To save time, teacher interviews were administered as 
written surveys. The Committee incorporated interviews of students and teachers from their feeder 
school in their data collection. Steering Committee members worked separately on their assigned 
report sections. Nonparticipation of non-language arts, content area teachers in the writing program 
emerged as the salient weakness in their program. The Committee convened a second faculty 
meeting for accomplishing the first part of step 4, setting priorities. Responsibility for the second 
part, planning actions, was given to a committee to be composed of the original Steering Committee 
plus a few other individuals. Before the end of the school year, the committee had incorporated new 
actions into their Consolidated Plan and hired a writing specialist. They agreed to complete planning 
during the summer before the start of the upcoming school year. 

The principal and two of the three Steering Committee members who responded to the 
feedback form believed that the outcome of their school study was worth the effort. They found the 
interviews to be the most helpful aspect of the study process. Steering Committee members spent 
from 3 1/2 -10 hours to complete the study (although the chair’s estimate seems to be low given the 
amount of work she did), report writing being the most time-consuming task. 



Independent School #3 

IS3 did not complete the school study to the knowledge of the research team. Initially, it 
seemed underway and was scheduled to be finished by April 5. The contact person first called the 
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school in January and called regularly thereafter to check its progress; she also visited the school 
twice and offered to help at various times. She initiated all contacts with the school. The assistant 
principal assured the contact that IS3 could progress unaided. There were apparently several 
extenuating circumstances that hindered the study’s completion, including dissension fi*om the 
SBDM Council. 



Independent School #4 

IS4 conducted its school study in six weeks with some successful results, although it had a 
slow start due to state assessment and faculty availability. The contact person spoke regularly to the 
principal before the study began; a few subsequent calls were not returned so she did not stay 
updated with the study’s progress. The contact person and Steering Committee chair did not speak 
at any time, so the school conducted its study rather independently. The Steering Committee chair 
replacement was given the position late in the school year. Therefore, she hurriedly organized and 
orchestrated a substantial amount of the early work herself, having the completed interviews ready 
for the other Steering Committee members at their first planning meeting. Staff fi*om the Family 
Resource Center interviewed teachers; higher grade level students from another school interviewed 
students; and the chair interviewed the administrators. The chair reported concerns regarding the 
student interviews such as interviewers not probing and interviewees not comprehending the 
questions. Faculty attended the rating meeting on a voluntary basis since it was held near the year’s 
end; slightly more than 50% of the faculty participated. The session was productive, according to 
the chair; they had time for discussion, but not for completing the school profile. The noticeable 
effect of the study was heightened awareness by nonaccountable teachers of the need for them to 
participate in writing instruction. Completing the profile and setting priorities were slated for the 
following school year due to lack of time. 

The principal and two of the three Steering Committee members believed that the school 
study was worth the effort. One was yet unsure. Committee members spent 6-11 hours each on the 
study, report writing taking the most time. They found the interviews and faculty discussion to be 
the most helpfiil aspects of the study. 



Independent Schuul #5 

IS 5 conducted its school study in six weeks late in the spring with some successful results, 
including germinal beginnings of a relationship with IS5's feeder school. Before the study began. 
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the contact person’s role changed to a facilitator’s role by the principal’s request. The 
contact/facilitator reported that the Steering Committee chair was initially uncertain of the study 
process and needed considerable assistance to get started. Also, the chair requested that the 
contact/facilitator perform all of the interviews because the Committee trusted her and knew that she 
was familiar with writing program jargon. Interviews from teachers and students at IS5's feeder 
school were included in IS5's data collection. The facilitator and scribe performed an unusually 
large number of student interviews (21) compared to the handbook recommendation (10-14). 
Moreover, they interviewed most students on an individual basis because the scribe conjectured that 
students would respond more openly in that format. The report was scheduled to be written in a 
group forum to include several teachers from both schools; however, the chair wrote most of the 
report herself The faculty rating meeting was planned for only faculty of the grades encompassed 
by the study from the two schools, instead of the entire faculty of IS5. Again, participation was a 
problem in that only two Steering Committee members besides the chair attended. By this time, 
several Steering Committee members had quit. 

According to the chair, the study results verified what they had always known to be their 
program’ s main weakness — inadequate writing instruction in nonaccountable grades; thus the study 
gave them the evidence they needed to enlist the participation of these teachers to create continuity 
in the writing program. The rating group intended to use its report of preliminary priorities as a 
“springboard” for joint discussion among teachers of certain grades from the two schools at the start 
of the next school year, hoping ultimately for regular joint planning time to align the curriculum. 
The principal and three Steering Committee members were unanimous that the school study was 
worth the effort. Steering Committee members invested 4-34 hours each in conducting the study, 
the chair doing much of the work herself The report-writing and rating tasks required the most time, 
and the interviews and dialogue at the rating meeting were the most helpful aspects of the study. 
Although step 4 of the study, setting priorities and planning actions, would not be finalized until the 
fall of the next year, the Steering Committee took initiative in engaging students in a promotional 
writing project during the summer. 



Independent School #6 

IS6 conducted its school study in nine weeks with some successful results. The Steering 
Committee requested clarification on one issue during the course of the study. The Steering 
Committee included five members, who also served as the interviewers and were allotted 1 month 
to complete the task of interviewing. A problem arose in interviewing students, because one of the 
interviewers gave the interview form to a few students to complete m written survey form; these 
forms contained many non-responses. Also, the chair was disappointed with the depth of the other 
student interviews. Steering Committee members worked separately on their assigned report 
sections, each allotted one week for this task. One third of the faculty attended the rating meeting. 
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which was concluded within two hours. Faculty did not engage in extended discussion after the 
rating activity was completed. Step 4 of the study was assigned to two committees in 
succession — first to the school’s writing committee, whose recommendations would be forwarded 
to the Consolidated Planning Committee for finalization. The principal and Steering Committee 
members all felt that their school study was worth the effort. The time investment of individual 
Steering Committee members ranged widely from 6-50 hours, interviewing requiring the most time 
of members. They also found the interviews to be the most helpful aspect of the study. 



Summary of the Results 

Ideal practices and counterintended practices in conducting the School Study have been 
discussed throughout the previous section of field-test findings. From these, best practices have been 
extrapolated and summarized in Table 4. The first section of the table lists guidelines that were 
already part of the handbook, but need to be reemphasized based on how important they proved to 
be or how weakly they were adhered to by field-test schools. The second section lists new and 
revised guidelines to be incorporated in the School Study handbook based on the field-test experience 
and, again, relative importance. 
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Table 4: Summary of Lessons Learned from the Field Test 
of the School Study of Writing Instruction 

Part I. Existing Guidelines to Reemphasize in the School Study of Writing Instruction 



Setup 



Emphasize the following recommendations: 

1. Gamer buy-in from faculty and inform Steering Committee members of the study’s purpose prior to its 
commencement. 

2. Ideally, the Steering Committee should include no more than 5 members. 



Interviews (step 1) 



Emphasize the following recommendations: 

3. It is preferable to solicit a pair of interviewers rather than only one. 

4. Interviewers should be parents or individuals not involved with regular operations of the school and 
relatively unknown to interviewees. Avoid committing faculty or other regular staff, whether on the 
Steering Committee or not, to this task. Higher grade level students from another school are an option, but 
not an ideal choice. 

5. Interviewers, particularly those unfamiliar with the writing program, need to be trained. 

6. You may interview more teachers and students than called for in the selection charts if you wish, but keep 
in mind that doing so will require you to spend more time assembling data into the report. 

7. Identify an appropriate district administrator interviewee. Only if that person is adamantly opposed to 
being interviewed should you ask that person to complete the interview in writing. 

8. Random selection of student interviewees is important and necessary. 

9. Interviewing students in groups of 2-3 is much preferable to interviewing them individually. DO NOT 
interview students via written survey. 

10. Probe student interviewees to elaborate on their answers. Rephrase questions if they do not understand 
what you are asking. 

1 1 . Compile teacher and student interviews. 



Report (step 2) 



Retain the following feature: 

12. A comprehensive sample report (a new one) — but exclude the brief sample report. 



Ancillary tasks 



Emphasize the following recommendation: 

13. Enlist clerical helpers for typing interview compilations and the report and for large copying jobs 



continued on the next page 
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Part II. Additional and Revised Guidelines to Include in the 
School Study of Writing Instruction 

Setup 

Include the following features: 

1. A revised time estimate for completing the School Study, i.e., 5-8 weeks 

2. A section of facilitator guidance notes, which would include tips on helping schools to handle such issues as facult> 
buy-in, diplomacy in report writing, and uncooperativeness of district administrators 

3. A section of guidelines/agendas for 2-4 Steering Committee planning meetings 

Include the following recommendations: 

4. Begin by early February to avoid conflicts with events occurring in late spring. 

5. Consider including the feeder school or next level school in your study. 

6. Refrain from having one Steering Committee member do a substantially greater amount of work than the others. 
The chair may have to do a little more work than others but not major steps unaided. 

Interviews (step 1) 

Delete the following option and include a recommendation against it: 

7. The option for teachers to complete the interview via written survey 

Include the following feature: 

8. Tips on gaining an interview with reluctant administrators 

Report (step 2) 

Include the following option and features: 

9. The option to write the report in a group meeting of the Steering Committee 

10. A “Table of report sections and data” that links interview questions to report topics for easier writing 

11. A revised time estimate for report writing, i.e., 3-5 hours per contributor 

Rating (step 3) 

Include the following recommendations or options: 

12. Invite faculty to the rating meeting on an expectant rather than a voluntary basis. 

13. If you wish, provide an entire rating guide to each meeting participant. 

Make the following revision: 

14. Identify indicators missing from the rating tools and correct the numbering so that they are synchronous to the 
report. 

Priorities and planning (step 4) 

Redesign this step: | 

15. Combine the first part of step 4 into step 3, i.e., transform the faculty rating meeting into both a rating and setting! 
priorities meeting, at which the school profile is used as a discussion piece. Make the second part of step 4,j 
planning, a supplemental section in the handbook. Thus “step 4” will no longer exist. 

Include the following recommendations: 

16. Progress to planning and action in a timely fashion after the rating/priorities meeting. 

17. If desired, repeat the study annually or as needed to measure writing program progress. 
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Conclusions 

Overall, positive conclusions are drawn about the self-study process, mainly based on 
schools’ extent of completion, respondents’ estimation ofits worth, and anticipated outcomes. Most 
schools completed the study to a satisfactory point and most Steering Committee participants 
believed that conducting the School Study of Writing Instruction (AEL & KDE, 1 999) was worth the 
effort. The time commitment required to undertake the study remains an issue of concern. 
Facilitator assistance was found to be highly beneficial. The School Study instrument was shown 
to be generally valid and replicable, but may be improved with various adjustments. 



The results of the field test indicate that for most schools and individual participants, the 
School Study of Writing Instruction was a valuable and successful experience. Most schools 
completed the study to a satisfactory interim point or to its conclusion. Most progressed to 
productive faculty discussion and preliminary priority work at the rating meeting. Before the close 
of 1 998-99, about half of the schools had already begun step 4, planning actions, and by the summer, 
two schools had acted on one of their new plans. Schools that had not begun planning had at least 
assigned the responsibility to a designated committee or the entire faculty for a later time. All 
schools expressed the intention to complete priority-setting and planning before or during the 
commencement of the 1999-2000 school year. It was promising that independent schools were able 
to conduct their study in about the same amount of time as facilitated schools. In sum, most schools 
demonstrated some degree of progress in decision making about enhancing their writing program, 
thus showing the School Study ’s capacity as a change agent. 

Several concrete outcomes emanated from schools’ self-studies. In response to a feedback 
form question about the effects and benefits of having conducted the School Study, only 7 of 48 
principal and Steering Committee member respondents anticipated no effects or benefits (five of 
these respondents being from one school, FS2). The 41 who responded otherwise listed many 
anticipated outcomes, some which were specific. The most frequently mentioned effects were (1) 
awareness of discontinuity in writing programs across grades and subject areas and, accordingly, (2) 
anticipated or already enacted measures to reverse that pattern. Other enacted or anticipated 
outcomes included revisions to school policies, changes in personnel, and new professional 
development. 

Steering Committee participants demonstrated that they were cognizant of a key mechanism 
of the study — that of comparing interview responses among role groups — in that the great majority 
named the interviews as the most helpful part of the School Study. Their appreciation for the 
interviews was further demonstrated by the frequent practice of including more individuals than the 
suggested number in the interviewee samples. This was surprising because the inclusion of more 
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interviews required more time to synthesize the data into the report, already the most time- 
consuming task of the study. In addition, some Steering Committee members found the rating 
meeting to be a valuable part of the study, demonstrating their appreciation for another key 
dynamic — ^that of all faculty sharing in writing program awareness. 

In addition to the conclusion that the School Study is valuable for schools, based upon the 
evidence of productive outcomes as discussed above, it is also established that the School Study was 
perceivedhy Steering Committee participants to be valuable, based on their assessment of its worth 
relative to the effort it required. A sizable majority of principals and Steering Committee members 
said that the school study was worth the effort, representing a majority in 9 of the 10 schools. If 
Steering Committee members had not deemed the School Study to be worthwhile, its impact may 
have been mitigated during the planning step; however, since many participants valued it, it is 
possible that planned interventions may proceed to full implementation. 

Despite these favorable results, it should be acknowledged that the study’s value may be 
attenuated for some schools by the time required to conduct it. Steering Committee members 
invested a substantial amount of time and work over a period of several weeks, among competing 
commitments and sometimes without personal remuneration, to complete their school study. (Yet 
schools’ delay in starting the study compounded the time pressures by forcing the study into an 
already busy period. Also, few schools effectively enlisted the aid of helpers for typing interviews 
and reports.) That Steering Committee members made these time sacrifices and still deemed the 
study to be worthwhile is a testament to the dedication of many teachers, administrators, and parents. 

Another attenuating factor was the method of distributing the $1,500 stipend. The stipend 
was intended to be greinted to schools after they had completed the school study and returned their 
feedback forms to the collaborative research team; however, most schools did not understand this 
and spent time securing it at the beginning of the study, a few because they needed it to pay for 
substitute teachers. The possibility of remuneration to future schools to aid them in implementing 
the study merits consideration. 

FS2's singularity of having a negative experience may be attributed to a combination of 
several possible reasons, hypothesized as follows. First, FS2 seemed to lack the desire to conduct 
the study at the outset, perhaps partially due to deficits in leadership. The Steering Committee chair 
did not weint the position, which was essentially forced on her. Also, the principal’s support was not 
evident. Second, FS2 was one of two schools that conducted the school study in the shortest time 
frame (five weeks) and the Steering Committee members reported that they felt rushed. Third, FS2 
had more individuals who spent a substantial number of hours (>20 hours) than any of the other 
schools; 4 of the 1 1 individuals in the field test who spent this number of hours were from FS2. The 
chair was one of these individuals, investing 30-40 hours in time-intensive tasks such as 
interviewing, compiling interviews, and typing the report. Fourth, FS2's Steering Committee 
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members were not compensated for their time because they used the stipend to pay for substitutes. 
It is difficult to ascertain whether this circumstance contributed to FS2's low evaluation of the School 
Study, because it is unknown how the majority of schools used their stipend in comparison (except 
that FS3 also used it to hire substitutes and IS5 divided it among Steering Committee members). 

In summary, short-term benefits of the School Study have been established, while long-term 
effects have yet to be proven. A later phase in the Kentucky writing project is plaimed to assess 
changes in writing scores of schools that have undertaken the School Study. At the least, the research 
team believes that because the School Study is based on already existing criteria of the Kentucky 
writing program, it is an exemplary formative evaluation tool for schools. Even if they cannot 
demonstrate the degree of improvement in writing scores over the next few years that they desire, 
they will be better equipped to ascertain why not and knowledgeably direct improvement efforts. 
Moreover, they will be able to show that they have made attempts at meeting accountability 
standards and perhaps undergird requests for additional resources based on systematic data that 
establish genuine areas of need. Yearly reviews of a school’s writing program, as suggested by a few 
principals in the field test, would further reinforce these benefits. 



Facilitator Assistance 

The field test demonstrated that facilitator assistance was advantageous for schools that 
received it. Independent schools conducted the School Study as efficaciously as facilitated schools 
in some respects and less so in other respects. The practices of both types of schools highlighted 
areas of need for facilitation in future schools. 

Based on the consideration of several outcomes, the collaborative research team concluded 
that facilitator assistance was beneficial and preferable to no facilitator assistance: 

• Steering Committees of FSs reported needing and appreciating the facilitator services 
that they received. The services that facilitators rendered alleviated Steering Committee 
chairs from duties associated with spearheading the school study, such as orchestrating 
and leading meetings and introducing the School Study process. The three services that 
they appreciated the most, in descending order, were leading the introductory meeting 
of the Steering Committee, assistance in interviewing, and assistance in report writing. 

• Steering Committee members at some ISs did request facilitative services from their 
contact person. One independent school evolved into a facilitated school by virtue of the 
amount of assistance that it requested from its contact person. 

• Some Steering Committee chairpersons at other ISs (who did not request facilitative- 
level services from their contact person) had to assume more responsibilities than their 
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counterparts at facilitated schools presumably because they did not have the services of 
a facilitator. Although Steering Committee chairs at both FSs and ISs performed many 
of the same tasks, it was clear that chairs at ISs also had to perform tasks similar to those 
performed by the facilitators at FSs; with a facilitator, they most likely would have been 
relieved of these extra duties. Essentially, some Steering Committee chairs at ISs 
became quasi-facilitators. 

• Also, both FSs and ISs engaged in some counterintended practices for which they could 
have benefited from further guidance, many related to the interviewing step. Three 
practices that were more evident in independent schools than in facilitated schools were 
(1) the failure to divide Steering Committee duties evenly among members, (2) the 
failure to involve the entire faculty in the rating meeting, and (3) the tendency to 
interview students individually. Yet FSs, despite facilitator assistance, experienced 
difficulties in implementing the School Study as well. These data, rather than mitigate the 
potential benefits of facilitator assistance, highlight areas where assistance can be 
improved. Facilitators can be trained to redirect some of the errant practices identified 
in the course of this field test. 

In summary, facilitator assistance proved to be valuable in several ways and possesses the potential 
for being so in future schools that undertake the School Study. 

Although facilitator assistance is deemed beneficial and desirable, it is true that schools 
perceived that they did not need considerable assistance. On the whole, both facilitated and 
independent schools initiated contacts with their facilitator or contact person minimally, and most 
did not request services beyond those already offered and provided to them. When asked about the 
amount of help required, FS respondents believed more than IS respondents that they needed 
facilitator assistance, but two thirds nevertheless maintained that they could have accomplished the 
study unaided. Surprisingly, independent schools respondents listed nothing when asked for which 
tasks they needed more assistance than was available. Participants from both types of schools 
demonstrated (or believed) that they were self-efficacious in conducting the School Study to the 
extent that they were compelled to be so. Still, the ability to be self-sufficient does not undermine 
the merits of facilitator assistance as established earlier; indeed, schools should be expected to be 
somewhat capable of performing a self-study, just as facilitators can be expected to enhance the 
process by which one is done. 

A few paradoxes arose to the conclusion that facilitator assistance is preferable and 
advantageous. Two paradoxes were that independent schools were more likely to initiate their study 
without prompting and a few independent schools progressed further in the study steps than 
facilitated schools by actuating the planning phase. A plausible explanation for these effects is that 
ISs relied on their own motivation because they basically had no one else to rely on. Also, largely 
self-guided with mainly the handbook at their disposal, ISs had no impetus for pausing, thus some 
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proceeded until all steps as outlined in the handbook were accomplished. Whereas in FSs, 
participants relied on the facilitator to prompt and direct them. The facilitator’s presence through 
the rating meeting step may have fostered the implicit assumption that the rating meeting was a 
natural breakpoint or concluding activity, temporarily drawing attention away from succeeding steps. 
However, evidence of greater efficiency by some ISs to begin and conclude the study need not negate 
the several advantages of having a facilitator’s aid at various junctures in the course of a school 
study. 



Another paradox was that one FS completed the study with an unfavorable impression 
despite having a facilitator’s assistance (and one IS did not complete the study despite the contact 
person’s offers to help at a level akin to facilitator assistance). This result may reveal more about 
capacity and motivation of some schools than the usefulness of facilitation. It is conceivable, for 
instance, that some low-functioning schools, already beset with a host of problems that contribute 
to their low performance, would lack the prerequisite motivation to conduct a self-study, when 
perhaps they could benefit most from one. Therefore, such schools may need intensive 
facilitation — initial capacity-building work and strong advocacy of the study’s benefits to raise 
faculty buy-in to an acceptable level — prior to launching the School Study. To these schools, the 
study would need to be presented well enough so that it is accepted at least as a necessity if not as 
a desirable. Then, the intensive facilitator might provide more than the usual amount of help for the 
time-intensive tasks of interviewing and report writing. Also, this type of facilitator could be 
instrumental in ensuring that the work is distributed advantageously among Steering Committee 
members as well as typists and interviewers not on the Committee. In a sense, intensive facilitation 
would be a way of infusing a school with temporary capacity so that the School Study could be a 
benefit rather than a burden. 



Validity and Reliability 

The reliability of schools’ reports as judged from the examination was inconclusive — some 
reports were representative of interview data and some were not. Only two of the seven reviewed 
would be considered model reports (and perhaps a third report as well, however a rigorous 
examination could not be performed on it). For the most part, reports were abbreviated and devoid 
of consistent discriminating detail. 

The deficiency in detail may be attributed to several factors. (1) It seems that report writers 
sometimes considered only one interview data item for report topics as if there were a 1:1 
relationship between them, while ignoring other relevant data items. Indeed, responses to report 
topics that required consideration of fewer data items were more likely to be consonant with the 
reviewer’s responses, whereas responses to more abstract or comprehensive topics were less likely 
to be consonant. Other hypothesized and stated reasons for concise and undetailed reports include 
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(2) limited time, (3) assumption of a certain level of prior knowledge by report readers (thereby 
obviating the writers from recording an undue amount of detail), (4) reluctance to write so pointedly 
that a particular person(s) or group is singled out for criticism (e.g., writers’ unwillingness to state 
in report topic 9C that not all faculty required writing portfolios), and (5) reluctance to repeat data 
recorded earlier in the report. 

That brevity and generality seem to have been the prevailing mode for reports necessitates 
inquiry about the utility of the report, i.e., is it reasonable to expect schools to produce model reports, 
and, do the outcomes of a school study depend that much on the representativeness of the report? 
First, given such conditions as limited time, writers’ reluctance to criticize others, and their 
assumption of readers’ prior contextual knowledge of the school, it is probably unreasonable to 
expect future schools to write comprehensive reports. More realistically, report writing will continue 
to be regarded as one of several tasks to be accomplished as efficiently as possible to enable progress 
to next steps in the study. Second, if this shall be the case, whether brevity affects school study 
outcomes depends on other aspects of the study. The report is the springboard to rating and, 
ultimately, faculty discussion and outcomes. If abbreviated summary statements are adequate for 
stimulating productive discussion and outcomes, then brevity in the report should not be a major 
concern; some precision in data transposition may be responsibly sacrificed. Indeed, facilitators or 
Steering Committee chairs at most schools indicated that faculties arrived at meaningful discussion 
despite perceptions of an unrepresentative report (or interviews). Also, most Steering Committee 
members asserted that the resultant school profile was reflective of their writing program, and they 
anticipated productive changes to the program as a consequence. 

On the other hand, a report without detail will deprive non-Steering Committee faculty of the 
essence of the interviews, the product that Steering Committee participants overwhelmingly 
identified as the most helpfiil aspect of the study — because they powerfully illustrated role groups’ 
perceptions of the writing program. Hopefully, the reinsertion in the final handbook of a “Table of 
report sections and data” that links interview questions to report topics will encourage writers to be 
more precise about how they incorporate the interview data into their reports. 

Validity and reliability should be entertained in the broader context of the School Study of 
Writing Instruction as an overall process, essentially a reconsideration of the study’s value. The 
research team has posed two questions to that end. One, has the independent variable (the School 
Study) caused an effect on the dependent variable (the writing program)? Two, can other schools 
also conduct the School Study of Writing Instruction with a similar measure of success? 

The team’s response to the first question is that immediate effects have been 
observ'ed — Steering Committees found the process to be informative and outcomes were 
implemented or anticipated. Yet it is too early to say what the ultimate effect will be. Year five of 
the Kentucky vsTiting project plans call for assessing the writing scores of schools that have 
completed the School Study of Writing Instruction and comparing them to scores from earlier years. 
The ultimate effect may depend on which priorities schools choose to act on, since some corrective 
actions may have more impact than others. 
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The team’s response to the second question is that other schools will be able to successfully 
conduct a School Study and find it valuable. Most field-test schools efficaciously conducted the 
study as outlined, at least to a logical interim point of completion, and most participants felt that the 
process was worthwhile. Another supporting argument for favorable replication is that field-test 
schools customized the School Study process in several ways — such as by writing the report in a 
group forum, interviewing more students or teachers than the minimum, involving a district writing 
leader on the Steering Committee, rating as a whole group if warranted by faculty size, beginning 
priority work during the rating meeting, and assigning planning to more than one entity. As well as 
contributing to improvements in the School Study handbook, these adaptations demonstrated the 
School Study’s malleability to individual school needs, evoking Berman and McLaughlin’s (1975) 
concept of mutual adaptation (Hord, 1987). Finally, the School Study is a targeted study, designed 
around 36 indicators of successful writing programs that were identified and replicated in many 
Kentucky schools. Thus, it provides an incentive for schools to evaluate their program by an 
established set of criteria. Yet at the same time, because most of the indicators are not exclusive to 
Kentucky’s writing program, the study is equally applicable to schools across the nation. 



Recommendations 

The following recommendations are based on the conclusions drawn in this report. They are 
intended to guide implementation decisions of the Kentucky Department of Education and AEL, Inc. 

Recommendation 1. Based on favorable outcomes in most field-test schools, the School 
Study of Writing Instruction is recommended for ongoing implementation in Kentucky schools. 
Performing this study promotes the practices of faculty collaboration in school instruction and of 
making informed decisions based on systematically collected data. Its use also reinforces the 
importance of including the voices of all school stakeholders, especially students, in school change 
efforts. 



Recommendation 2. Given the time required to conduct the School Study of Writing 
Instruction, the Kentucky Department of Education and local administrators should consider 
supporting schools that undertake the study with stipends and/or release time. Also, schools should 
be urged to begin the School Study no later than the first of February to avoid competing demands 
on faculties’ time. 

Recommendation 3* Facilitator assistance is beneficial and advantageous for maximizing 
schools’ use of the School Study. It is recommended that external facilitators be identified, trained, 
and made available to schools in the fall of 1999. The availability of facilitators will be an 
opportunity for introducing the School Study to schools in addition to being an enhancement to its 
implementation. 
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Recommendation 4. Although the School Study is an optional intervention, KDE should 
consider taking steps to identify and encourage low-performing and continuously declining schools 
to undertake it, which might include providing intensive facilitation beyond the usual level. 

Recommendation 5. The collaborative research team should monitor the progress of future 
schools that undertake the School Study. The team should continue with plans to assess the changes 
in writing scores of participating schools in Year five of the Kentucky writing project. 

Recommendation 6. A reconfigured “Table of report sections and data” should be included 
in the final School Study of Writing Instruction handbook to increase the representativeness of 
schools’ reports. 

Recommendation 7. All other revisions to the School Study handbook as listed in Table 4 
should be made to enhance its overall usability and representativeness. These revisions include a 
modified time estimate for conducting the School Study, modified time estimates for certain study 
steps; changes to the orchestration and procedure of certain study steps, mainly the interviews; 
redesigns to some of the steps; and alignment of indicators between various forms. 

Recommendation 8. The School Study of Writing Instruction should be developed as a 
stand-alone product, not only for Kentucky writing programs, but for writing programs nationwide. 

Recommendation 9. The research into indicators of successful writing programs as the basis 
for the School Study of Writing Instruction may be considered a model for development of other 
curricular needs assessment instruments (AEL, 1997a; AEL, 1997b; Coe et ah, 1999a, 1999b). Not 
all of this research base is specific to writing instruction; some parts are generic to teaching, learning, 
and school efficacy. Thus the School Study may be adapted easily as a self-study needs assessment 
for content areas other than writing. 
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APPENDIX A 

Observed Score Differences, t-Values, p-Values, and Significance Values of All Indicators 
Between the Continuously Improving and Continuously Declining Schools 
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W.ai is the task? : 



rime and Task S ummary 



Hovv long do(^ it take? 



Who docs it?:: 



Select people to be interviewed, 
schedule interviews, select and, if 
necessary, train interviewers 


? Time depends on ease of 
recruiting interviewers and on 
whether training is necessary. 


Steering committee 


Complete an interview about 
writing instruction 


About one hour for each interview 


Four to 8 teachers and their 
interviewers (persons from outside 
the school) 


Interview a random sample of 8-12 
students (total) from the 
accountable grade, previous and 
following grades. 


About four hours 
Students can be interviewed in 
groups of two or three. Each 
interview takes 30 to 45 minutes. 


Students and interviewer(s) (person 
or persons from outside the 
school.) 


Gather data about resources 
available to the school’s writing 
program 


About one hour 


Principal or his/her^designee 


Interview principal and district 
administrator 


One hour each 


Steering committee member(s) 


Develop a summary report, using 
survey results, resource inventory 
and 

student interviews. 


One to two hours per person 


Steering committee members and 
additional people, if needed 


Make copies of the report for all 
faculty members. 


One to two hours. 


Support staff persons. 


Prepare to lead faculty meeting to 
assess writing program 


One hour 


Steering committee 


Using the report, assess school’s 
strengths and needs on the analysis 
form provided. 


Three hours 


Entire faculty led by the steering 
committee'-^ 


(Optional) 

Prepare to facilitate faculty meeting 
to choose improvement priorities. 
Copy materials needed in meeting. 


One hour 


Steering comminee 


(Optional) 

Decide on improvement priorities. 


Three hours 

. 


Steering committee and faculty 



'The faculty will work in small groups on different sections of the analysis form. This approach deepens the 



faCUltV*S UnderSTandmp nf writino nmor?>m’c ^ 



iwds and increases coirmiitrneni to improvements. 



Schools that are so large that it is unworkable to include all faculty in this meeting can select a representative 
sample of teachers from each department or grade. A group of 1 8 to 24 people is adequate to complete the rating 

task. However, try to include as many teachers as possible in order to build understanding of the writing nrosram 
across the faculty. & & 
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